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Abstract 



We study associative memory of an oscillator neural network with distributed 
native frequencies. The model is based on the use of the Hebb learning rule 
with random patterns (£f = ±1), and the distribution function of native fre- 
quencies is assumed to be symmetric with respect to its average. Although 
the system with an extensive number of stored patterns is not allowed to 
get entirely synchronized, long time behaviors of the macroscopic order pa- 
rameters describing partial synchronization phenomena can be obtained by 
discarding the contribution from the desynchronized part of the system. The 
oscillator network is shown to work as associative memory accompanied by 
synchronized oscillations. A phase diagram representing properties of memory 
retrieval is presented in terms of the parameters characterizing the native fre- 
quency distribution. Our analytical calculations based on the self-consistent 
signal-to-noise analysis are shown to be in excellent agreement with numerical 
simulations, confirming the validity of our theoretical treatment. 



I. INTRODUCTION 

Most of the theoretical models for associative memory of neural networks as typified 
by the Hopfield model have been based on the idea of rate coding, which assumes that 
information is coded in the firing rate of a neuron at a particular time. On the other hand, 
some experimental results on the visual cortex have been suggesting the possibility of another 
scheme of information coding, that is, the concept of temporal coding which assumes that 
information is coded in the relative timing of the firing pulses 0. 

Stimulated by these experimental findings, studies of the temporal coding are becoming 
an active area of theoretical brain researches. Recently, there have been an increasing number 
of papers reporting neurophysiological experimental findings of synchronization phenomena 
of neurons in animal's brain ||. Synchronized firings of neurons can be considered to play 
a key role in certain types of information processing such as the binding problem |4j]. 

In order to get insights into basic properties of the scheme of the temporal coding it will 
be necessary to study neural network models based on neurons that can be expressed by 
such nonlinear oscillators as limit-cycle type and integrate-and-fire type oscillators. Indeed, 
a network of integrate-and-fire neurons, where one can be directly concerned with the spike 
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timing of neurons, is one of such models and was previously studied || to confirm its 
functioning as associative memory accompanied by synchronized firings. 

The simplest theoretical model of synchronization of coupled oscillators will be the case 
of a system of phase oscillators of Kuramoto et al. which is derived under the as- 

sumption of weak coupling limit for general types of limit-cycle oscillators. In an associative 
memory model based on such a system with the Hebb type learning rule, the distribution of 
native frequencies will become crucial for the behavior of synchronization or desynchroniza- 
tion of the system [|G|,|9|-|lT|j • If all of the oscillators have an identical native frequency and the 
couplings are symmetric, the network can be reduced, by an appropriate coordinate trans- 
formation, to a system that has a Lyapunov function ensuring stability of its equilibrium 
state(s), which implies that the network eventually settles into a synchronized state. On 
the other hand, in the case when native frequencies are distributed, Lyapunov functions in 
general no longer exist except for a particular case |12] and the network becomes essentially 
of dynamic nature; one may expect partial synchronization or more complex oscillatory 
behavior. 

In the present paper, we report results of our detailed analysis of associative memory of 
an oscillator neural network model with distributed native frequencies. Whereas oscillator 
networks with a Lyapunov function were studied previously by several authors 1 13,13 1, sat- 



isfactory analysis of the case without a Lyapunov function has been far less conducted [15 



II. ANALYSIS OF THE MODEL 

The oscillator network model we consider is described by the following phase evolution 
equations 0: 

^ = Ui-J2 JijSin((j>i- (pj), (i = l,---,N) (1) 

where N denotes the total number of neurons, and <pi and u>i respectively denote the phase 
and the native frequency of i-th neuron. We assume the strength of interactions to be given 
according to the Hebb learning rule, 

^ = ^E^, (2) 
n 

where {£f } (fi — 1- • -p) represents the /i-th stored patterns. We consider the case when the 
number of patterns is extensive, p = aN. In order to elucidate the effect of the distribution 
of native frequencies on the behavior of synchronization we want to make the model as 
simple as possible, and then we assume £f to take values ±1 rather than continuous values 
with £f = exp(i#f), (9f = [0 : 2n]), which would be more appropriate for the study of 
temporal coding itself. 

Furthermore, we deal with a simple case where the native frequency distribution p(uS) is 
discrete and symmetric with respect to the central frequency cu : p{oj) = Y, k =-L C k 5(tu — cu k ), 
u k + u_ k = 2cu , C k = C_fc > 0, J2k=-L Cfc = 1- It is noted that without loss of generality 
one can set ujq = as a result of rotational symmetry of the phase evolution equation ([]]). If 
the native frequencies are the same, i.e., Ui = Q independent of i, all of the oscillators can 
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get synchronized with (j>i(t) = 0° + fit. In the case of distributed native frequencies, there 
remain, in general, a group of desynchronized oscillators. Then the total system cannot 
settle into entirely equilibrium states. 

Assuming that influences exerted by a group of desynchronized neurons on the macro- 
scopic behavior of the total system can be neglected, we use the self-consistent signal-to-noise 
analysis (SCSNA) [[31| to analytically obtain the macroscopic order parameter equations for 
the time-stationary states of synchronized oscillations of the network accompanying memory 
retrieval. 

Introducing the complex variable x^ = exp(i0fc) to express the state of k-th neuron, we 
formally obtain the fixed point equations by putting — 0; 



+ \f\hi 



Xi = , \hi\ > u h (3) 

hi 

where hi = J2j Jij x j is the local field and h% denotes its complex conjugate. Defining the 
overlap 

^ = ^E^. ( 4 ) 

i 

one can rewrite the local field as hi = Y^n^i^- 

Within the frame work of the SCSNA, considering the retrieval solutions of Eq. (|]) for 
which m 1 = m ~ 0(1) and m M ~ 0(l/y/N)(fj, ^ 1), we assume the local field to be described 

as 
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hi = ilm + Zi + TiXi + T 2 Xi. (5) 

Here the first term involving m is the signal part, while the remaining terms represent noise 
part involving complex Gaussian noise Zi = Ui + ivi (u^ real) together with the effective 
self-coupling terms proportional to Xi and its complex conjugate Xj. We note that Eq. ([|) 
yield no solutions if \hi\ < which does not ensure \xi\ = 1. This means that neurons with 
\hi\ < Ui cannot take part in synchronized motions exhibited by neurons with \hi\ > uj{. 
Although the desynchronized neurons, each of which is expected to oscillate with a certain 
modified frequency, will make the local fields time dependent quantities, their effect can be 
expected to cancel out to good approximation provided taking the time average is considered. 
In further analysis, we discard the contribution from the desynchronized neurons by setting 
Xi = for \hi\ < Ui. 

Following the standard procedure of the SCSNA, we obtain the self-consistent equations 
for macroscopic variables in the limit N — » oo. Making use of the rotational symmetry of 
the phase evolution equation Eq. ([!]) to choose a gauge such that overlap m becomes real 
i.e. [JDuDv ^sin^]^ ^ = 0, we have 



m 



jj^2& x i(£ii u h Zi,Zi) = J DuDu^cos 

i 

? = ^E( Re N) 2 = [/d«d 



(6a) 



)v(cosd>) 2 



(6b) 
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with u, u, v ) being implicitly determined by 

= — uj + (^m + u) sin <p — v cos + T 2 sin 20 = 0. 



(6c) 

(6d) 

(6e) 
(6f) 
(6g) 
(6h) 

(6i) 
(7) 



In the above equations, [ £i} means taking the average over the distribution p(u>) and 
the pattern j^ 1 }, and the gaussian integration DuDv is to be performed over the noise u, v 
satisfying the condition \hi\ > u^. It is noted that in performing the gaussian integration in 
Eqs. (6a)- (6i), one has to take into account the Maxwell rule to pick up the relevant solution 
<fii, when Eq. (0) admits multi-solutions owing to the presence of the self-coupling term (r 2 
term). Unlike T 2 term, T\ term has no contribution to the equilibrium fixed-point equation 
(7). 

In what follows, for the sake of simplicity we deal mainly with the case 



p(u) = C 5(uj) 



Co., n . 1 - Cc 
- — d{uj - uji) H — 



5(u + ui), (L = 1, ujq = 0). 



(8) 



A characteristic feature of the distribution is that the presence of oscillators with central 
frequency u is allowed with a finite fraction C and the effect of desynchronized part can 
be described in terms of u)\ and C alone. 

Setting uji = recovers the case that allows an energy function that is bounded from 
below, E[{(pi}] = -\ J2ij Jij cos(0j - fa). 

Then all of the oscillators get synchronized for large times with the equilibrium configura- 
tion {0,} determined by Eq. (|3|) or Eq. (|7]) together with the Maxwell rule, which is explained 
in Fig. 1. Analysis based on the SCSNA of such networks as having Lyapunov functions is 
on the same level of approximations as the replica symmetric calculations [|16]-[18|. Figure 2 
represents the dependence of m on the loading rate a computed from the SCSNA equations 
(6) and (|7|). We see that the storage capacity is given by a c = 0.0395 with m c = 0.68. The 



present value of the storage capacity slightly differs from the result reported previously |]14 
which was obtained by an inappropriate treatment of the Maxwell rule JL£j|J . We note that 
the Maxwell rule becomes necessary only for a in the neighborhood of a c . Under the as- 
sumption of random patterns with £f = ±1, the magnitude of T 2 plays a crucial role in 
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determining the a c , as can be seen in Fig. 2. This should be compared with the case of 
£f = exp(i#f) (0f = [0 : 2n]), where the order parameter equations involve no T 2 term as in 
the case of Cook's model |13[ (see also recent work in Ref. We note here that when 



discrete patterns with Q-state variables 0f = 2imf /Q, (nf = 0, • • • , Q — 1) are considered, 
the T 2 term does not appear in the order parameter equations, if Q > 3. 

The effect of distribution of native frequencies on the behavior of memory retrieval is 
summarized in Fig. 3, where overlap m from the SCSNA equations is plotted as a function 
of a and oj\ for C = 0.7. There appear two distinct retrieval regimes separated by a 
valley or gap located around a region with an intermediate value of u)\ in the m-a-Ui space. 
In the regime with small oui, most of the oscillators undergo synchronized motions. The 
storage capacities a c corresponding to the edge of the m surface are observed to decrease, 
as ui is increased, to attain a certain minimum value, where a crossover to the other regime 
occurs. In the regime with large u>i, oscillators with uj\ oscillate with their own frequencies 
that are modified from the original u)\ as a result of entrainment phenomenon. Such a 
behavior can easily be understood by considering the system in the limit u\ — > oo. In the 
large u)\ limit, while most of the oscillators with Ui = ujq get synchronized, the oscillators 
with Ui = ±oui cannot get synchronized and they oscillate with their own frequency ±a>i 
independently of the synchronized neurons with Ui — Uq. It is clearly seen in this case that 
the desynchronized neurons does not contribute to the time-averaged local fields acting on 
the synchronized neurons. Then the system can be viewed as a diluted system with only 
a fraction Co of neurons participating in memory retrieval. It is interesting to note that 
the storage capacity a c in the large oj\ region increases as iO\ is increased. The crossover 
between the two regimes or the gap in the m[io\) curve with fixed a can be more clearly seen 
in Fig. 4, which display the iO\ dependence of overlap m obtained from the SCSNA together 
with the result of numerical simulations (N = 2000) for a fixed value of a. 

As the fraction Co of the oscillators with the central frequency u>o is varied, there occur 
three types of behavior of the ra(a>i) curve showing the cross over between the two regimes. 
In Fig. 5, we show the U\ dependence of the overlap m obtained for Co = 0.5, 0.7 and 0.9 with 
a = 0.02. While for Co large one sees a continuous crossover between the small o>i regime 
and the large iO\ regime, for Co small the large u)\ regime disappears. For only intermediate 
values of Co, one observes the gap in the m{uj\) curve to appear. 

The occurrence of such two retrieval regimes is not restricted to the simplest case of 
L = 1, but is also observed for more general case with L > 2. When a sufficiently large 
number of oscillators have native frequencies near the central one uq, the system can behave 
the same way as that of L = 1 with C ^ 0, as is shown Fig. 6, where the co> 2 dependence 
of the overlap m obtained by numerical simulations is displayed for the native frequency 
distribution with L = 2 and Co = 0: 

1 — 2Ci 1 — 2Ci 
p(u) = Ci5(uj - uoi) + C\5(uo + ui) H 5(lu - uj 2 ) H 5(u + u 2 ). (9) 

Indeed we see a gap separating the small 002 regime from the large cc>2 one. Although there 
are no oscillators with u^o = because of Co = 0, most of the oscillators get synchronized 
for the small 002 regime. 



5 



III. SUMMARY AND DISCUSSIONS 



We have studied the behavior of the oscillator neural network system with distributed 
native frequencies. To date analysis of associative memory models has been mostly confined 
to the case of networks with an energy or Lyapunov function that allows use of replica 
symmetric calculations of statistical mechanics. In the present work, making use of the 
method of the SCSNA that is free from the energy concept fj.2(|, we have succeeded in 
analyzing properties of the associative memory accompanied by synchronized oscillations 
in the prototype oscillator network that has no Lyapunov functions except for the case 
when p(uj) = 5(u — Uq). We have shown that the oscillator network can work as associative 
memory based on temporal coding of simple type even in the presence of distribution of native 
frequencies. Our approach has taken advantage of the fact that such temporal attractors 
as limit-cycles of certain types of dynamical systems can be reduced to a fixed-point type 
attractors as a result of the system's symmetry property. 

The distribution of native frequencies does not allow the present coupled oscillator system 
to settle into an entirely synchronized state but into a partially synchronized one. The 
contributions from desynchronized neurons to the macroscopic behavior of the system have, 
however, been found to be almost negligible. Thereby, the partially synchronized state 
of the system has turned out to be almost determined by the long time behavior of the 
group of synchronized neurons, which can be described by fixed-point type attractors giving 
rise to retrieval states. In other words, memory retrieval is achieved by synchronization of 
oscillatory motions of neurons. 

Under the assumed type of native frequency distribution we have found that the partial 
synchronization is classified into a high degree of synchronization that occurs for small U\ 
with overlap m large and a low degree of synchronization that occurs for large LO\ with m 
small. 

Finally we note that oscillator neural networks are considered to have advantage over 
fixed-point type neural networks in several respects. First, oscillator neural networks exhibit 
the ability to easily and efficiently discriminate a successful retrieval from unsuccessful one, 
because the settling into a retrieval state of oscillator neural networks implies the appearance 
of oscillations with an appreciable amplitude and the central frequency u> in the overlap, 
and hence, the local fields of neurons. 

Second, by utilizing phase as well as amplitude as dynamical variables representing out- 
put of a neuron |[21] it becomes possible for neural information to be processed in terms of 
spatio-temporal patterns of neuronal firings. In particular, information on time domain is 
available by employing the scheme of temporal coding |ll]JT3|JT5|j2^1 , where assuming uni- 
formly distributed random numbers #f on [0, 2tt) for components of the memory patterns, 
the phase difference fa — <f>j between the two oscillators i and j eventually settles into the 
difference 6% — 6j of the memory pattern \i. Our present model setting based on the use 
of binary patterns with 6% = 0, 7r [[L4]] presents a special as well as simple case of the 



above temporal coding, where the pattern of synchronization is either of in phase or out of 
phase. Such a case may be related to the problem of segmentation of an object from its 
background |23|-p5|. 

More generally, pattern segmentation p6[ seems to be one of the unique features of 



oscillator neural networks as has been studied by Wang et al. [27 , who pointed out that use 
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of limit cycles as attractor states of associative memory facilitates switching between one 
pattern and another on the time domain. 

While studying the important issue of the functionality of pattern segmentation provided 
by oscillator neural networks will require taking into account some specific ingredients such 
as the sparseness of the memory patterns and appropriate inhibitory couplings, our simple 
model will have wide applicability in exploring the computational ability or relevance exhib- 
ited by oscillator neural networks from the viewpoint of analytical studies. Extending the 
assumed symmetric native frequency distribution to more general cases of asymmetric one 
as well as continuous ones is straightforward. Results of such issues together with details of 
the present work will be reported elsewhere. 
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FIGURE CAPTIONS 



FIG.l: Maxwell rule used in the SCSNA for picking up the relevant solution for output Xi as a 
function of u and v among multi-solutions to Eq. (|7]). In a graphical representation of 
solving Eq. (|7|) for which three solutions (given by the intersections of the two curves) 
appear as u or v is varied, the relevant solution is chosen as the external intersection 
that delimits the larger area enclosed between the two curves. Such solution is marked 
by the filled circle. The Maxwell rule ensures the condition of the free energy min- 
imum [17| in a system with a Lyapunov function as in the case of the liquid-vapor 



phase transition. 

FIG. 2: a dependence of the overlap m obtained from the SCSNA (solid line) together with 
that from numerical simulations with N = 4000 in the case p(oo) = S(u — ujq). To 
observe the result that the contribution of the I^ term in Eq. (|J) or (|7|) to the value of 
m is significant, we plot values of m obtained by deliberately setting I^ = (broken 
line). 

FIG. 3: Phase diagram showing m-surface plotted as a function of a and u)\ for the network 
with distributed native frequencies withp(u;) = 0.75(u) + 0.155(u — uj\) +0.155(u + u 1 ). 
We observe a valley or gap separating the small cut regime from the large uu% one. The 
dependence of the storage capacities a c on ui is represented by the projected curve on 
the a-Ui plane. 

FIG. 4: Dependence of m on iO\ for the network with a = 0.02 and Co = 0.7 obtained from the 
SCSNA (line) and numerical simulations with iV = 2000 (dots). The gap separating 
the two regimes with different types of synchronization are clearly depicted and is in 
excellent agreement with the results of numerical simulations. 

FIG. 5: Dependence of m on uoi for a = 0.02 and Cq = 0.5, 0.7, 0.9 obtained from the SCSNA. 

FIG. 6: The numerical simulation result (N = 4000) for L = 2 showing the co>2 dependence of 
m in the case with u = 0, C = 0, oj\ = 0.1, C\ = 0.35 and a = 0.02 (see eq. (§)). 
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